Method and apparatus for performing query aware partitioning

ABSTRACT

A method and system for providing query aware partitioning are disclosed. For example, the method receives a query plan comprising a plurality of queries, and classifies each one of the plurality of queries. The method computes an optimal partition set for each one of the plurality of queries, and reconciles the optimal partition set of each one of the plurality of queries with at least one subset of queries of the plurality of queries. The method selects at least one reconciled optimal partition set to be used by each query of the plurality of queries, and stores the selected at least one reconciled optimal partition set in a computer readable medium.

The present invention relates generally to partitioning high-rate datastreams and, more particularly, to a method and apparatus for queryaware partitioning of high-rate data streams.

BACKGROUND OF THE INVENTION

Data Stream Management Systems (DSMS) are gaining acceptance forapplications that need to process very large volumes of data in realtime. Applications such as network monitoring, financial monitoring,sensor networks and the processing of large scale scientific data feedsproduce data in the form of high-speed streams. Data streams arecharacterized as an infinite sequence of tuples that must be processedand analyzed in an on-line fashion to enable real-time responses. Theincreasing use of DSMSs has led to their use for ever more complex querysets.

The load generated by such applications frequently exceeds by far thecomputation capabilities of a single centralized server. In particular,a single-server instance of a DSMS, e.g., Gigascope, cannot keep up withthe processing demands of new networks, which can generate more than 100million packets per second.

SUMMARY OF THE INVENTION

In one embodiment, the present invention provides a method and systemfor providing query aware partitioning. For example, the method receivesa query plan comprising a plurality of queries, and classifies each oneof the plurality of queries. The method computes an optimal partitionset for each one of the plurality of queries, and reconciles the optimalpartition set of each one of the plurality of queries with at least onesubset of queries of the plurality of queries. The method selects atleast one reconciled optimal partition set to be used by each query ofthe plurality of queries, and stores the selected at least onereconciled optimal partition set in a computer readable medium.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 depicts a high-level block diagram of an exemplary architecturefor query aware partitioning according to one embodiment of the presentinvention;

FIG. 2 depicts a flow diagram of a method for query aware partitioningaccording to one embodiment of the present invention;

FIG. 3 depicts an additional flow diagram of a method for query awarepartitioning; and

FIG. 4 illustrates a high level block diagram of a general purposecomputer suitable for use in performing the functions described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

It is to be noted, however, that the appended drawings illustrate onlyexemplary embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

DETAILED DESCRIPTION

FIG. 1 depicts a high-level block diagram of an exemplary architecture100 for query aware partitioning according to one embodiment of thepresent invention. In one embodiment, the architecture comprises anetwork 102, a partitioning and query optimization module 104, amonitoring module 106, one or more nodes 108 ₁ to 108 _(n), a terminalnode 112 and one or more data streams 110 ₁ to 110 _(n). The network 102may be any type of a network, such as for example, a local area network(LAN), a wide area network (WAN), an intranet, an internet and the like.

In an illustrative embodiment, the monitoring module 106 includes one ormore query plans 1 to N. The query plans include instructions on how thedata streams 110 ₁ to 110 _(n) should be processed by nodes 108 ₁ to 108n. For example, the query plans may include a combination of queries ina query language (e.g., standard query language (SQL)) to execute theprocessing of data. The query plans may including instructions on how toassign an operator to each node of the plurality of nodes 108 ₁ to 108_(n), parameters of each of the assigned operators for each of theplurality of nodes 108 ₁ to 108 _(n), instructions as to how data fromthe data streams 110 ₁ to 110 _(n) should be distributed among nodes 108₁ to 108 _(n) and informing each node of the plurality of nodes 108 ₁ to108 _(n) a source and a destination of a data stream 110 ₁ to 110 _(n)that a node of the plurality of nodes 108 ₁ to 108 _(n) will process.

Those skilled in the art will recognize that operators are programmingelements of a query. For example, an operator may be join, select,merge, aggregate and the like. The types of operators compatible withthe present invention are not limited by the examples provided above ordiscussed herein. Any type of operator may be used.

The parameters for the operators may include the predicates within aparticular query. For example, the parameters for a selection operatorin a selection query may be a predicate of LENGTH=5 and a selection list(i.e. the desired data or output) such as a source IP address. Furtherexamples of parameters are provided with respect to the exemplaryaggregation and join queries discussed below.

The monitoring module 106 may be in communication with the partitioningand query optimization module 104. The partitioning and queryoptimization module receives the data streams 110 ₁ to 110 _(n). Thepartitioning and query optimization module is also in communication withthe one or more nodes 108 ₁ to 108 _(n). The partitioning and queryoptimization module may be implemented within a server or computerhaving a processor, input output devices and memory (not shown). In oneembodiment, the monitoring module 106 and the partitioning and queryoptimization module 104 may be located within the same device, forexample a server of a computer, or be located on separate devices, forexample separate servers or separate computers.

The nodes 108 ₁ to 108 _(n) are used to execute an optimized query plan,as discussed below. Although only one level of nodes 108 ₁ to 108 _(n)is illustrated in FIG. 1, it should be noted that there may beadditional levels of nodes to execute various levels of the optimizedquery plan. Moreover, one or more of the nodes 108 ₁ to 108 _(n) mayreside on a single host (not shown). In other words, there may be one ormore hosts in network 102 and each host may have one or more of thenodes 108 ₁ to 108 _(n).

Ultimately, the data may be forwarded to a terminal node 112 for finalprocessing. The terminal node 112 may output or display the finalresults of the optimized query plan to a user, another node or themonitoring module 106. Alternatively, the terminal node 112 may storethe output of the final processing of data in a computer readable mediumfor later retrieval or use.

When a user desires to monitor or gather a particular data set (alsoreferred to herein as tuples) within one or more of the data streams 110₁ to 110 _(n), the partitioning and query optimization module 104 mayobtain an appropriate query plan from the monitoring module 106.

Currently, the query plan may be applied to the data streams 110 ₁ to110 _(n) by brute force. As a result, the processing capabilities ofcurrent network architectures are unable to apply the queries andprocess the data to execute the query plan in an efficient manner.

The present invention provides a novel query aware partitioning methodprovided by the partitioning and query optimization module 104. Giventhe appropriate query plan from monitoring module 106, the partitioningand query optimization module 104 may calculate an optimal partitioningset to transform the query plan provided by monitoring module 106 intoan optimized query plan. As a result, the one or more nodes 108 ₁ to 108_(n) coupled to the partitioning and query optimization module 104 mayexecute the optimized query plan. An exemplary method for calculatingthe optimized query plan is discussed below with reference to FIG. 2.

FIG. 2 depicts a flow diagram of a method 200 for query awarepartitioning according to one embodiment of the present invention. Inone embodiment, method 200 may be executed by the partitioning and queryoptimization module 104.

The method 200 begins at step 202 and proceeds to step 204. In step 204,the method 200 receives a query plan comprising a plurality of queries.As discussed above, the query plan may be received by the partitioningand query optimization module 104 from the monitoring module 106. Inaddition, the query plan may comprise a plurality of queries. A group ofdifferent queries with different operators may be used to construct aquery plan, which will obtain a desired set of data from the datastreams 110 ₁ to 110 _(n).

Those skilled in the art will recognize how to construct various queriesof different operators such as aggregate queries, join queries, selectqueries and the like. For illustration, a few formats of various queriesare provided herein. For example, an aggregation query may have theformat:

-   -   SELECT tb, srcIP, destIP, sum(len)    -   FROM PKT    -   GROUP BY time/60 as tb, srcIP, destIP        The SELECT predicate indicates selecting to report the fields        tb, srcIP, destIP and sum(len) where tb represents time bucket,        srcIP represents a source IP address, destIP represents a        destination IP address and sum(len) represents the sum of all        values in the length field for unique values of tb, srcIP and        destIP of a data stream. The FROM predicate identifies a source        of the data stream, in this case data stream PKT. The GROUP BY        time bucket indicates that the data will be grouped by tb of        time/60, srcIP and destIP.

In another example, a join query may have the format:

-   -   SELECT time, PKT1.srcIP,    -   PKT1.destIP, PKT1.len+PKT2.len    -   FROM PKT1 JOIN PKT2    -   WHERE PKT1.time=PKT2.time and    -   PKT1.srcIP=PKT2.srcIP and    -   PKT1.destIP=PKT2.destIP        The SELECT predicate indicates selecting time, PKT1.srcIP,        PKT1.destIP, PKT1.len+PKT2.len, where PKT1.srcIP represents the        source IP address from data stream PKT1, PKT1.destIP represents        the destination IP address from data stream PKT1,        PKT1.len+PKT2.len represents the sum of the length of data        selected from data streams PKT1 and PKT 2. The FROM predicate        represents the sources of the data streams that will be joined,        for example PKT1 and PKT2. The WHERE predicate represents where        time of data stream PKT1 is equal to time of data stream PKT2        and similarly for source IP address and destination IP address        for data streams PKT1 and PKT2. Although only two types of        queries are detailed above, those skilled in the art will        recognize that the syntax for additional queries may be derived        from the examples provided above.

At step 206, method 200 classifies each one of the plurality of queriesfound in the query plan. For example, the query plan may comprise acombination of select queries, aggregation queries, join queries and thelike.

At step 208, method 200 computes an optimal partition set for each oneof the plurality of queries. The computation of an optimal partition setdetermines a distribution plan for incoming data from data streams 110 ₁to 110 _(n) that maximizes the amount of data reduction that can bepreformed locally before transporting the intermediate results to a nodethat produces final results, e.g., terminal node 112.

For example, in one embodiment the optimal partitioning set foraggregation queries may be defined as follows:

-   -   SELECT expr₁, expr₂, . . . , expr_(n)    -   FROM STREAM_NAME    -   WHERE tup_predicate    -   GROUP BY temp_var, gb_var₁, . . . ,    -   gb_var_(m)    -   HAVING group_predicate        In an optimal partitioning set for an aggregation query, only a        subset of the group by variables (gb_var₁, . . . , gb_var_(m))        that can be expressed as a scalar expression (expr₁, expr₂, . .        . , expr_(n)) involving an attribute of one of the source input        streams (STREAM_NAME) are considered. As a result, Lemma 1 may        be defined as follows:    -   Lemma 1. Let G be a set of group-by attributes referenced by the        query Q and let P be portioning set, P=(sc_expr(attr₁),        sc_exp(attr₂), . . . sc_exp(attr_(n))). Query Q is compatible        with partitioning set P if and only if for any pair of tuples        tup1 and tup2 G(tup1)=G(tup2)        P(tup1)=P(tup2).        Following Lemma 1, any compatible partitioning set for        aggregation query Q will have the form {sc_exp(gb_var₁), . . . ,        sc_exp(gb_var_(n))}, where sc_exp(x) is any scalar expression        involving x. Given that there are an infinite number of possible        scalar expressions, every aggregation query has an infinite        number of compatible partitioning sets. Furthermore, any subset        of a compatible partitioning set is also compatible.

In another example, the optimal partitioning set for join queries may bedefined as follows:

-   -   SELECT expr₁, expr₂, . . . , expr_(n)    -   FROM STREAM1 AS S    -   {LEFT|RIGHT|FULL} [OUTER] JOIN    -   STREAM2 AS R    -   WHERE STREAM1.ts=STREAM2.ts    -   and STREAM1.var_(1l)=STREAM2.var_(2l)    -   and . . . STREAM1.var_(1k)=    -   STREAM2.var2 _(k) and other_predicates;        For ease of analysis, only join queries whose WHERE clause is in        Conjunctive Normal Form (CNF) in which at least one of the CNF        terms is an equality predicate between the scalar expressions        involving attributes of the source streams are considered. In an        optimal partitioning set for a join query, let J be a set of all        such equality predicates {sc_exp(R.rattr₁)=sc_exp(S.sattr₁), . .        . , sc_exp(R.rattr_(n))=sc_exp(S.sattr_(n))}. As with        aggregation queries, only scalar expressions involving        attributes of the source input streams are considered. Join        queries that do not satisfy these requirements are considered as        incompatible with any partitioning set. As a result, Lemma 2 may        be defined as follows:    -   Lemma 2. Let J be a set of equality join predicates of the query        Q and let P be portioning set, P=(sc_expr(attr₁), sc_exp(attr₂),        . . . sc_exp(attr_(n))). Query Q is compatible with partitioning        set P if and only if there exists a non-empty subset J′ of        J s. t. for any pair of tuples tup1 from R and tup2 from S s. t.        J′ is satisfied        P(tup1)=P(tup2).        Following Lemma 2, the partitioning sets for two streams S and R        using Partn_R={sc_exp(R.attr₁), . . . , sc_exp(R.attr_(n))} and        Partn_S={sc_exp(S.attr₁), . . . , sc_exp(S.attr_(n))},        respectively, can be computed. It also follows that the join        query is compatible with any non-empty subset of its        partitioning set. Since it is not feasible to partition the        input stream simultaneously in multiple ways, Partn_R and        Partn_S will need to be reconciled to compute a single        partitioning scheme, which will be discussed below. Those        skilled in the art will recognize that optimal partitioning sets        for other queries, such as for example, union queries, select        queries and the like, may be derived from the optimal        partitioning set examples for aggregation and join queries        provided above.

At step 210, the method 200 reconciles the optimal partition set of eachone of the plurality of queries with at least one subset of queries ofthe plurality of queries. Once an optimal partition set for each one ofthe plurality of queries is computed, the optimal partition sets much betested against all other queries and subset of queries within the queryplan to ensure compatibility. This process is referred to herein asreconciling the optimal partition sets.

Reconciling the optimal partition sets may generate a new grouping setcompatible with another query or subset of queries. This new groupingset may be referred to as Reconcile_Partn_Sets( ), defined as follows:

-   -   Def. Given two partitioning set definitions PS1 for query Q1 and        PS2 for query Q2, Reconcile_Partn_Sets( ) is defined to return        the largest partitioning set Reconciled_PS such that both Q1 and        Q2 are compatible with partitioning using a Reconciled_PS. The        empty set is returned if no such Reconciled_PS exists.

Considering a simple case of partitioning sets consisting of just datastream attributes (i.e. no scalar expressions involved),Reconcile_Partn_Sets ( ) returns the intersection of the two partitionsets. For example, Reconcile_Partn_Sets({srcIP, destIP}, {srcIP, destIP,srcPort, destPort}) is the set {srcIP, destIP}. For a more general caseof partitioning sets involving arbitrary scalar expressions,Reconcile_Partn_Sets uses scalar expression analysis to find a “leastcommon denominator”. For example, Reconcile_Partn_Sets({sc_exp(time/60), sc_exp(srcIP), sc_exp(destIP)}, {sc_exp(time/90},sc_exp(srcIP & 0xFFF0)}) is equal to a set {sc_exp(time/180,sc_exp(srcIP & 0xFFF0)}. The Reconcile_Partn_Sets function can make useof either simple or complex analysis based on the implementation timethat is available.

At step 212, the method 200 selects a reconciled optimal partition setto be used by each query of the plurality of queries in the query plan.For example, the selected reconciled optimal partition set may beselected based on a compatibility and lowest cost computation.

In one embodiment, computing a compatible partitioning set for anarbitrary query plan essentially requires reconciling all therequirements that all nodes in the query graph place on compatiblepartitioning sets. A simplified implementation of the procedure ofcomputing compatible sets PS for a Directed Acyclic Graph (DAG) with nnodes would be as follows:

-   -   1. For every query node Q_(i) in a query DAG, compute the        compatible partitioning set PS(O_(i)).    -   2. Set PS=PS(Q_(i)).    -   3. For every iε[1 to n], set PS=Reconcile_Partn_Sets(PS,        PS(Q_(i)).        Although many realistic query sets result in the partitioning        set PS to be empty due to conflicting requirements of different        queries, a reasonable approach is to try to satisfy a subset of        nodes in a query DAG in order to minimize the total cost of the        query plan. There are a variety of different cost models that        can be used to drive the optimization.

In one exemplary cost model, the cost model defines a cost of the queryplan to be the maximum amount of data a single node 118 ₁ to 118 _(n) inthe query plan is expected to receive over the network 102 during onetime epoch. This model tries to avoid query plans that overload a singlenode 118 ₁ to 118 _(n) with excessive amounts of data.

Let R be the rate of an input stream 110 ₁ to 110 _(n) on which thequery set is operating, and PS be a partitioning set. For each querynode Qi in a potential query execution plan we define the followingvariables:

-   -   selectivity_factor (Qi). The selectivity factor estimates the        expected ratio of the number of output tuples to the number of        input tuples Qi receives during one epoch.    -   out_tuple_size (Qi). Expected size of the output tuple produced        by Qi.    -   recursively define input_rate (Qi) to be R if Qi is a leaf node        and to be the sum of all output_rate (Qj) s.t. Qj is a child of        Qi.    -   output_rate (Qi)=(input_rate (Qi)/in_tuple_size        (Qi))*selectivity_factor (Qi)*out_tuple_size (Qi).        The cost(Qi) is defined in the following way:    -   0 if it processes only local data.    -   input_rate (Qi) if Qi is incompatible with PS.    -   output_rate (Qi) if Qi is compatible with PS.        The intuition behind this cost formula is that an operator        partitioned using a compatible partitioning set only needs to        compute the union of the results produced by remote nodes, and        therefore the rate of the remote data it is expected to receive        is equal to its output rate.

Finally, we define the cost of the query plan Qplan given partitioningPS cost(Qplan, PS) to be the max cost(Qi) for all i. The goal of thisformula is to prevent overloading a single node rather than minimizingaverage load.

With the above cost model, an optimal reconciled portioning set may beselected at step 212 based upon compatibility and lowest cost. A methodfor computing a lowest cost takes a query DAG as an input and produces apartitioning set that minimizes the cost of the query plan. The methodenumerates all possible compatible partitioning sets using dynamicprogramming to reduce the search space. An outline of the method is asfollows:

-   -   1) For every query node Q_(i) in a query DAG, compute its        compatible partitioning set PS(i) and cost(Qplan, PS(i)). Add        non-empty PS(i) to a set of partitioning candidates.    -   2) Set PS to be PS(i) with minimum cost(Qplan, PS(i)).    -   3) For every candidate pair of partitioning sets PS(i) and PS(j)        compute compatible partitioning set PS(i,        j)=Reconcile_Partn_Sets(PS(i), PS(j)) and cost(Qplan, PS(ij)).        Add non-empty PS(i, j) to a set of candidate pairs.    -   4) Set PS to be PS (i, j) with minimum cost(Qplan, PS(l, j)).    -   5) Similarly to previous step, expand candidate pairs of        partitioning sets to candidate triples and compute corresponding        reconciled partitioning sets and minimum cost.    -   6) Continue the iterative process until we exhaust the search        space or end up with an empty list of candidates for the next        iteration.        Since it is impossible for a partitioning set to be compatible        with a node and not to be compatible with one of the node        predecessors, the following heuristics can be used to further        reduce the search space:    -   Only consider leaf nodes for a set of initial candidates.    -   When expanding candidate sets only consider adding a node that        is either an immediate parent of a node already in the set or is        a leaf node.

At step 214, the method 200 stores the selected at least one reconciledat least one partition set in a computer readable medium. For example,the computer readable medium may be a hard drive disk, a read onlymemory (ROM), a random access memory (RAM), floppy disk drive, or anyother data storage device. The selected at least one reconciled at leastone partition set may then be retrieved and applied to the query plan asdescribed below with reference to FIG. 3.

FIG. 3 depicts an additional flow diagram of a method 300 for queryaware partitioning according to one embodiment of the present invention.In one embodiment, method 300 may also be executed by the partitioningand query optimization module 104.

The method 300 begins at step 302 and proceeds to step 304. At step 304,the method 300 applies the selected reconciled optimal partition set,from method 200, to the query plan to transform the query plan into anoptimized query plan. In one embodiment, the optimized query plan has aplurality of optimized queries that are executed in accordance with theselected reconciled optimal partition set. The optimized query plandistributes data received from at least one data stream 110 ₁ to 110_(n) to a plurality of nodes 108 ₁ to 108 _(n) in accordance with theselected reconciled optimal partition set.

In one embodiment, transforming the query plan into an optimized queryplan comprises two phases. The first phase is to build apartition-agnostic query plan. Let S be a partitioned source input datastream consumed by a query set, S=∪Partn_(i). A partition-agnostic queryplan is created by creating an additional merge query node that computesa stream union of all the partitions and making all query nodes 108 ₁ to108 _(n) that consume S read from the merge node. Since each host mighthave multiple CPUs/Cores, multiple partitions may be allocated to eachparticipating host depending on the host capabilities.

The second phase is to perform query plan transformation in a bottom-upfashion. All transformation rules that are used for partition-relatedquery optimization consist of two procedures: Opt_Eligible( ) andTransform( ). Opt_Eligible( ) is a Boolean test that takes a query nodeand returns true if it is eligible for partition-related optimization.Transform( ) replaces the node that passed Opt_Eligible( ) test byequivalent optimized plan. The pseudo code for query optimizer is givenbelow:

-   -   1) Compute a topologically sorted list of nodes in the query DAG        Q₁, Q₂, . . . , Q_(n) starting with the leaf nodes.    -   2) For every i ∈ [1 to n]        -   If Opt_Eligible(Q_(i))            -   Transform(Q_(i), Partitiong_Info)

Performing the transformation in a bottom-up fashion allowstransformation compatible leaf nodes to be easily propagated through thechain of compatible parent nodes. A detailed description of theimplementation of Opt_Eligible( ) and Transform( ) for aggregationsqueries and join queries are discussed below. The present transformationmethods developed for aggregation queries and join queries can beapplied to simpler queries such as selection queries, merge queries,projection queries and the like.

For transformation of aggregation queries, the Opt_Eligible( ) procedurefor an aggregation query Q and partitioning set PS returns true if thefollowing conditions are met:

-   -   Query Q has a single child node Mof type merge (stream union).    -   Each child node of M is operating on single partition consistent        with PS.    -   Q is compatible with PS.    -   Q is the only parent of M.        The last requirement is important to prevent the optimizer from        removing the merge nodes that are used by multiple consumers.

In a transformation for compatible aggregation query nodes, the mainidea behind the Transform( ) procedure for eligible aggregation query Qis to push the aggregation operator below a merge M and allow it toexecute independently on each of the partitions. For each of the inputsof M a copy of Q can be created and pushed below the merge operator. Inthis embodiment, data is fully aggregated before being sent to aterminal node 112 that does not require any additional processing.

In a transformation for incompatible aggregation queries (i.e.aggregation queries that fail the Opt_Eligible( ) test), options arestill available that perform better than the default partition-agnosticquery execution plan. The idea behind the proposed optimization is theconcept of partial aggregates. This idea may be illustrated on a querythat computes a count of number of packets sent between pairs of hosts:

-   -   Query tcp_count:        -   SELECT time, srcIP, destIP, srcPort, COUNT(*)        -   FROM TCP        -   GROUP BY time, srcIP, destIP, srcPort

The tcp_count can be split into two queries called sub- andsuper-aggregate:

-   -   Query super_tcp_count:        -   SELECT time, srcIP, destIP, srcPort, SUM(cnt)        -   FROM sub_tcp_count        -   GROUP BY time, srcIP, destIP, srcPort        -   Query sub_tcp_count:        -   SELECT time, srcIP, destIP, srcPort, COUNT(*) as cnt        -   FROM TCP        -   GROUP BY time, srcIP, destIP, srcPort

All the SQL's built-in aggregates can be trivially split in a similarfashion. Many commonly used User Defined Aggregate Functions (UDAFs) canalso be easily split into two components. Note that all the predicatesin the query's WHERE clause can be pushed to sub-aggregates, but allpredicates in the HAVING clause need complete aggregate values and,therefore, must be evaluated in super-aggregate.

For transformation of join queries and other multi-way join queries, theOpt_Eligible( ) procedure for a join query Q and partitioning set PSreturns true if the following conditions are met:

-   -   Query Q has a two children nodes M1 and M2 of type merge (stream        union).    -   Each child node of M1 and M2 is operating on single partition        consistent with PS.    -   Q is compatible with PS.    -   Q is the only parent of M1 and M2.

The idea behind the Transform( ) procedure for an eligible join query Qis to perform pair-wise joins for each of partition of input stream.This is accomplished by creating a copy of join operator and pushing itbelow the child merges. The left side partitions that do not havematching right side partitions and similarly unmatched right sidepartitions are ignored for inner join computations. For outer joincomputations, unmatched partitions are passed through special projectionoperator that adds appropriate NULL values needed by outer join. Theoutput tuples produced by the projection operator are then merged withthe rest of the final results.

At step 306, the method 300 analyzes the at least one data stream 110 ₁to 110 _(n) in accordance with the optimized query plan. For example,using the optimized query plan, a desired set of data from one or moreof the data streams 110 ₁ to 110 _(n) on may be obtained in a moreefficient way.

At step 308, the method 300 outputs a result of the analysis to a user.For example, as discussed above, the data may be transmitted to aterminal node 1 12 for final processing. The terminal node 112 mayoutput the data to a user by displaying the data on a display device.Alternatively, the data may be stored at the terminal node 112 forfurther analysis or may be forwarded to another node 118 ₁ to 118 _(n),the monitoring module 106 or the partitioning and query optimizationmodule 104. The method 300 concludes at step 310.

It should be noted that although not specifically specified, one or moresteps of method 200 and 300 may include a storing, displaying and/oroutputting step as required for a particular application. In otherwords, any data, records, fields, and/or intermediate results discussedin the method can be stored, displayed and/or outputted to anotherdevice as required for a particular application. Furthermore, steps orblocks in FIGS. 2 and 3 that recite a determining operation or involve adecision do not necessarily require that both branches of thedetermining operation be practiced. In other words, one of the branchesof the determining operation can be deemed as an optional step.

FIG. 4 depicts a high level block diagram of a general purpose computersuitable for use in performing the functions described herein. Asdepicted in FIG. 4, the system 400 comprises a processor element 402(e.g., a CPU), a memory 404, e.g., random access memory (RAM) and/orread only memory (ROM), a module 405 for query aware partitioning, andvarious input/output devices 406 (e.g., storage devices, including butnot limited to, a tape drive, a floppy drive, a hard disk drive or acompact disk drive, a receiver, a transmitter, a speaker, a display, aspeech synthesizer, an output port, and a user input device (such as akeyboard, a keypad, a mouse, and the like)).

It should be noted that the present invention can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a general purposecomputer or any other hardware equivalents. In one embodiment, thepresent module or process 405 for query aware partitioning can be loadedinto memory 404 and executed by processor 402 to implement the functionsas discussed above. As such, the processes provided by the module 405for query aware partitioning (including associated data structures) ofthe present invention can be stored on a computer readable medium orcarrier, e.g., RAM memory, magnetic or optical drive or diskette and thelike.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

1. A method for processing a query, comprising: receiving a query plancomprising a plurality of queries; classifying each one of saidplurality of queries; computing an optimal partition set for each one ofsaid plurality of queries; reconciling said optimal partition set ofeach one of said plurality of queries with at least one subset ofqueries of said plurality of queries; selecting at least one reconciledoptimal partition set to be used by each query of said plurality ofqueries; and storing said selected at least one reconciled optimalpartition set in a computer readable medium.
 2. The method of claim 1,further comprising: applying said selected at least one reconciledoptimal partition set to said query plan to transform said query planinto an optimized query plan; applying said optimized query plan to atleast one data stream; and outputting a result of said applying to auser.
 3. The method of claim 2, wherein said optimized query plan has aplurality of optimized queries and wherein said optimized query plandistributes data received from said at least one data stream to aplurality of nodes in accordance with said selected at least onereconciled optimal partition set.
 4. The method of claim 2, wherein saidapplying said optimized query plan comprises: assigning an operator toeach node of a plurality of nodes that each node will execute; providingone or more parameters for each operator at each of said plurality ofnodes; and informing each node of said plurality of nodes a source and adestination of said at least one data stream.
 5. The method of claim 1,wherein said reconciling step is repeated until all possible combinationof subsets of said plurality of queries have been exhausted.
 6. Themethod of claim 1, wherein said selected at least one reconciled optimalpartition set is selected based upon a lowest cost computation.
 7. Themethod of claim 6, wherein said lowest cost computation comprisesdetermining a reconciled optimal partition set to be used by each queryof said plurality of queries that provides a least amount of datatransfer between a plurality of nodes.
 8. The method of claim 1, whereinsaid reconciling said optimal partition set for each one of saidplurality of queries with at least one subset of queries of saidplurality of queries comprises: selecting an optimal partition set thatis compatible with at least two queries of said plurality of queries;and using said selected optimal partition set for at least said twoqueries of said plurality of queries.
 9. The method of claim 1, whereinsaid query plan is used for extracting data from a high-rate datastream.
 10. A computer-readable medium having stored thereon a pluralityof instructions, the plurality of instructions including instructionswhich, when executed by a processor, cause the processor to perform thesteps of a method for processing a query, comprising: receiving a queryplan comprising a plurality of queries; classifying each one of saidplurality of queries; computing an optimal partition set for each one ofsaid plurality of queries; reconciling said optimal partition set ofeach one of said plurality of queries with at least one subset ofqueries of said plurality of queries; selecting at least one reconciledoptimal partition set to be used by each query of said plurality ofqueries; and storing said selected at least one reconciled optimalpartition set in a computer readable medium.
 11. The computer-readablemedium of claim 10, further comprising: applying said selected at leastone reconciled optimal partition set to said query plan to transformsaid query plan into an optimized query plan; applying said optimizedquery plan to at least one data stream; and outputting a result of saidapplying to a user.
 12. The computer-readable medium of claim 11,wherein said optimized query plan has a plurality of optimized queriesand wherein said optimized query plan distributes data received fromsaid at least one data stream to a plurality of nodes in accordance withsaid selected at least one reconciled optimal partition set.
 13. Thecomputer-readable medium of claim 11, wherein said applying saidoptimized query plan comprises: assigning an operator to each node of aplurality of nodes that each node will execute; providing one or moreparameters for each operator at each of said plurality of nodes; andinforming each node of said plurality of nodes a source and adestination of said at least one data stream.
 14. The computer-readablemedium of claim 10, wherein said reconciling step is repeated until allpossible combination of subsets of said plurality of queries have beenexhausted.
 15. The computer-readable medium of claim 10, wherein saidselected at least one reconciled optimal partition set is selected basedupon a lowest cost computation.
 16. The computer-readable medium ofclaim 15, wherein said lowest cost computation comprises determining areconciled optimal partition set to be used by each query of saidplurality of queries that provides a least amount of data transferbetween a plurality of nodes.
 17. The computer-readable medium of claim10, wherein said reconciling said optimal partition set for each one ofsaid plurality of queries with at least one subset of queries of saidplurality of queries comprises: selecting an optimal partition set thatis compatible with at least two queries of said plurality of queries;and using said selected optimal partition set for at least said twoqueries of said plurality of queries.
 18. The computer-readable mediumof claim 10, wherein said query plan is used for extracting data from ahigh-rate data stream.
 19. An apparatus for processing a query,comprising: means for receiving a query plan comprising a plurality ofqueries; means for classifying each one of said plurality of queries;means for computing an optimal partition set for each one of saidplurality of queries; means for reconciling said optimal partition setof each one of said plurality of queries with at least one subset ofqueries of said plurality of queries; means for selecting at least onereconciled optimal partition set to be used by each query of saidplurality of queries; and means for storing said selected at least onereconciled optimal partition set in a computer readable medium.
 20. Theapparatus of claim 19, further comprising: means for applying saidselected at least one reconciled optimal partition set to said queryplan to transform said query plan into an optimized query plan; meansfor applying said optimized query plan to at least one data stream; andmeans for outputting a result of said applying to a user.